<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title>Amazon Mechanical Turk for LabelMe</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<link href="style.css" rel="stylesheet" type="text/css">
<script language="JavaScript" type="text/JavaScript">
<!--
function MM_reloadPage(init) {  //reloads the window if Nav4 resized
  if (init==true) with (navigator) {if ((appName=="Netscape")&&(parseInt(appVersion)==4)) {
    document.MM_pgW=innerWidth; document.MM_pgH=innerHeight; onresize=MM_reloadPage; }}
  else if (innerWidth!=document.MM_pgW || innerHeight!=document.MM_pgH) location.reload();
}
MM_reloadPage(true);
//-->
</script>
</head>


<!--<body style="width:53.5em;">-->

<table style="width:62em;" border="0">
  <tr>

    <td style="width:8em;" valign="top"><a href="index.html" target="_blank"><img style="width:8em;" src="Icons/mt_example_good4.jpg" border="0" /></a></td>

    <td valign="top" bordercolor="#000000">

<h1>Amazon Mechanical Turk for LabelMe</h1>

<p>
Want to outsource your labeling task to the internet?  <a href="http://mturk.com" target="_blank">Amazon
Mechanical Turk</a> allows access to many internet users who are ready to
perform tasks for a fixed price.
</p>

<p>
The idea is simple: you provide a task and a selling price.
Internet workers perform the task and are subsequently paid.
In Mechnical Turk terminology, tasks are called "HITs", people
requesting work are called "Requesters", and people who do the work
are called "Workers".
</p>

<p>
This page describes how to set up <a href="index.html" target="_blank">LabelMe</a>
annotation tasks onto Mechanical Turk.
The process is simple, as we have provided scripts for creating and
sending LabelMe annotation tasks onto Mechanical Turk.
All you have to do is follow the instructions below and pay workers on
Mechanical Turk to label images.  We collect the annotations, which
are immediately available for download.
In this way, everybody wins: Mechanical Turk workers get paid, you get
your images annotated, and the computer vision community gets access
to more hand-labeled data.
</p>


<h2>Instructions for setting up LabelMe on Mechanical Turk</h2>

<p>
Setting up LabelMe on Mechanical Turk is easy.  The following are
instructions for setting up LabelMe on Mechanical Turk.
</p>

<h3>1. Upload images onto LabelMe</h3>

<p>
To upload a collection of images to LabelMe, simply <a
href="AboutMe.html" target="_blank">contact us</a> and we can arrange to put your
images into a folder.
We ask that your images do not contain any spaces or special
characters (use <code>_</code> for space) and that the images have a
.jpg extension.  Your images will live at:
<!--Instructions for uploading images to LabelMe are <a href="http://people.csail.mit.edu/torralba/labelme/upload3D.html" target="_blank">here</a>.-->
</p>

<p>
<code>http://labelme.csail.mit.edu/Images/FOLDER</code>
</p>

<p>
Remember the name of <code>FOLDER</code> as this will be used later.
</p>

<h3>2. Set up an Amazon Mechanical Turk account</h3>

<p>
You will need to set up an account as a Requester on Mechanical
Turk.  Instructions for setting up an account are <a
href="https://requester.mturk.com/mturk/welcome" target="_blank">here</a>.  Once you
have created an account, sign in and try to access <a
href="https://requester.mturk.com/mturk/resources" target="_blank">your account</a>,
along with the <a href="https://requestersandbox.mturk.com/mturk/resources" target="_blank">sandbox</a> (used for debugging).
</p>

<h3>3. Download and install Amazon Mechanical Turk Command Line
Tools</h3>

<p>
You will need to install the Amazon Mechanical Turk Command Line
Tools.  The tools provide the backbone for communicating with the
Mechanical Turk servers.  To start, you first need to request your access key and secret
key.  This is different than your username and password.  To do this,
create an <a href="http://aws.amazon.com" target="_blank">Amazon Web Services
account</a>.  Once the account is created, go to "Your Account->Access
Identifiers", which is located at the top of the page.  Here, you will
find your access key and secret key.
</p>

<p>
Next, download the <a
href="http://developer.amazonwebservices.com/connect/entry.jspa?entryID=694&ref=featured" target="_blank">Amazon
Mechanical Turk Command Line Tools</a> (note that so far, we have only
tested the Linux/Mac version).
Unzip the file and follow the instructions inside the directory to install the
Command Line Tools.
</p>

<p>
As a reminder, open and modify
<code>./aws-mturk-clt-1.3.0/bin/mturk.properties</code> to include your access key and secret key.
Also, make sure you set the following environment variables
(e.g. using the command "<code>export VAR=/path/to/file</code>" in bash):
<br />
<br />a) $MTURK_CMD_HOME - this should point to the location of your
Amazon Mechanical Turk Command Line Tools (root directory).
<br />b) $JAVA_HOME - this should point to the location of your Java
installation.
<!--<br />
Also make sure that the $MTURK_CMD_HOME environment variable points to the
location of your Amazon Mechanical Turk Command Line Tools (root directory).
-->
</p>

<h3>4. Download scripts for sending LabelMe jobs to Mechanical
Turk</h3>

<p>
Scripts for sending LabelMe jobs to Mechanical Turk are available
<a href="LabelMeMechanicalTurk.zip">here</a>
(currently, the scripts are available for Linux/Mac only).
After download, unzip the file and change to the directory.
</p>

<p>
The zip file contains a set of scripts that are used to
interact with Mechanical Turk, along with files used to set how the
task is performed (e.g. how much do the workers earn, list of jobs,
etc.).
In addition, a Matlab function is included to assist in setting which
images are to be annotated.
</p>

<h3>5. Submit jobs to Mechanical Turk</h3>

<p>
Let us now submit jobs to Mechanical Turk.
In this section, we will use Mechanical Turk's "sandbox" server.
The sandbox is free to use (you have fake credits on this server), so
you can test here to make sure everything works.  When you are ready
for the real thing, simply remove all <code>-sandbox</code> flags and
set <code>sandbox=0</code> in the MATLAB script below.  Make sure to
add money to your Amazon account before running on the real server.
</p>

First, go to the directory
containing the scripts for sending LabelMe jobs to Mechanical Turk
(see Step 4 above; <code>cd /path/to/LabelMeMechanicalTurk/</code>).
Next, start MATLAB and run the following at the prompt (use the value
of <code>FOLDER</code> from Step 1 above):
</p>

<p>
<code>
folder=FOLDER;
<br />sandbox=1; // Set this to sandbox=0 when running on the real server
<br />generateLabelMeInputFile(folder,sandbox);
</code>
</p>

<p>
The script will produce a file called <code>labelme.input</code> with
all of your images listed as HITs.  The help output (type <code>help
generateLabelMeInputFile</code> at the MATLAB prompt) gives
information on further customizations for the task.
</p>

<p>
We are now ready to submit to Mechanical Turk.
To submit to the sandbox, run the following at the Linux command
prompt (not the MATLAB prompt):
</p>

<p>
<code>
./run.sh -sandbox
</code>
</p>

<p>
You will see messages indicating that the jobs are being submitted.
At the end, there will be a URL that points to a preview page for the
HIT.  You can go to that URL and try out the HIT.  All of the
collected annotations are stored on the LabelMe servers, so you can
<a href="instructions.html" target="_blank">download them immediately</a> or view
thumbnails of the images
(<code>http://labelme.csail.mit.edu/browseLabelMe/FOLDER.html</code>).
</p>

<p>
Mechanical Turk also produces outputs for the HITs, which you can
retrieve from the server.  To do this, run the following at the Linux
command prompt:
</p>

<p>
<code>
./getResults.sh -sandbox
</code>
</p>

<p>
This will produce a file called <code>labelme.results.xls</code>.
This is a comma separated file, which can be viewed with Excel.  This
file lists the status of each HIT, as well as outputs from the
annotation task, such as number
of annotations labeled, last object labeled, etc.
</p>

<p>
You can also view the status of HITs on the requester page of the <a
href="https://requester.mturk.com/mturk/manageHITs"
target="_blank">Mechanical Turk server</a> (or on the requester page
of the  <a href="https://requestersandbox.mturk.com/mturk/manageHITs"
target="_blank">sandbox server</a>).  Note that you can navigate to
the status page from the main requester page via the "Manage HITs
individually" link at the top right of the page.
</p>

<p>
It is important to pay the workers as soon as possible.  To pay the workers, run the following at the Linux
command prompt:
</p>

<p>
<code>
./reviewResults.sh -sandbox
</code>
</p>

<p>
Finally, when all of the HITs are completed, you can remove them from
the Amazon server by running at the Linux command prompt:
</p>

<p>
<code>
./approveAndDeleteResults.sh -sandbox
</code>
</p>

<p>
The file <code>labelme.properties</code> contains all of the pricing
settings for the HIT.  Change this file as you wish.  Currently, each
HIT is priced at $0.01 per image.  Please see the documentation for
the Amazon Mechanical Turk Command Line Tools for more information.
</p>

<h2>Sample results and cost considerations</h2>

<p>
The quality of the annotations provided by Mechanical Turk workers is
in general quite good.  The following are example annotations provided by the
workers:
</p>

<p>
<table>
<tr>
<td><img src="Icons/MTthumbs/img11.jpg" /></td>
<td><img src="Icons/MTthumbs/img10.jpg" /></td>
<td><img src="Icons/MTthumbs/img3.jpg" /></td>
</tr>
<tr>
<td><img src="Icons/MTthumbs/img4.jpg" /></td>
<td><img src="Icons/MTthumbs/img5.jpg" /></td>
<td><img src="Icons/MTthumbs/img1.jpg" /></td>
<!--<td><img src="Icons/MTthumbs/img9.jpg" /></td>-->
</tr>
<tr>
<td><img src="Icons/MTthumbs/img7.jpg" /></td>
<td><img src="Icons/MTthumbs/img8.jpg" /></td>
<td><img src="Icons/MTthumbs/img6.jpg" /></td>
</tr>
</table>
</p>

<p>
The following are statistics for the tasks that we submitted to
Mechanical Turk.
</p>

<p>
<table border="1">
<tr>
<td align="center"><b>Task</b></td>
<td align="center"><b>Price per image</b></td>
<td align="center"><b>Task description</b></td>
<td align="center"><b># images</b></td>
<td align="center"><b># annotations<br />collected</b></td>
<td align="center"><b>Time elapsed<br />(hours)</b></td>
<td align="center"><b># workers</b></td>
</tr>
<tr>
<td align="center">1</td>
<td align="center">$0.01</td>
<td align="center">Label at least one object in the image</td>
<td align="center">237</td>
<td align="center">678</td>
<td align="center">13.5</td>
<td align="center">37</td>
</tr>
<tr>
<td align="center">2</td>
<td align="center">$0.01</td>
<td align="center">Label at least five objects in the image</td>
<td align="center">271</td>
<td align="center">1492</td>
<td align="center">23.13</td>
<td align="center">43</td>
</tr>
<tr>
<td align="center">3</td>
<td align="center">$0.01</td>
<td align="center">Label as many objects as you wish in the image</td>
<td align="center">271</td>
<td align="center">627</td>
<td align="center">9.08</td>
<td align="center">28</td>
</tr>
</table>
</p>

<p>
We also received feedback from the workers.  The following are all of
the feedback from the workers:
</p>

<table border="1">
<tr>
<td align="center"><b>Positive</b></td>
<td align="center"><b>Negative</b></td>
<td align="center"><b>Other</b></td>
</tr>
<tr>
<td align="center">fine</td>
<td align="center">this very heavy work for this hit</td>
<td align="center">one of them is all messed up</td>
</tr>
<tr>
<td align="center">good</td>
<td align="center">$0.01 for five objects now?</td>
<td align="center">Sorry if anything is miss spelt.</td>
</tr>
<tr>
<td align="center">Very interesting idea</td>
<td />
<td />
</tr>
<tr>
<td align="center">fun trying this</td>
<td />
<td />
</tr>
<tr>
<td align="center">No feedback.  This was fun!</td>
<td />
<td />
</tr>
</table>

<p>
It seems in general that workers enjoy performing the task.  In addition,
sometimes they provided additional useful feedback on the task
(e.g. workers would see polygons that other labelers had provided; one
commented on the quality of a polygon provided by another worker).
</p>

<p>
The following paper provides additional information for labeling tasks
on Mechanical Turk:
</p>

<p>
A. Sorokin and D. Forsyth.  <a href="http://www.cs.uiuc.edu/homes/sorokin2/papers/cvpr08_annotation.pdf" target="_blank">Utility data annotation with Amazon
Mechanical Turk</a>.  First IEEE Workshop on Internet Vision at CVPR,
2008.
</p>

<h2>Let us know!</h2>

<p>
We are very excited about the annotation possibilities using Mechanical Turk
with LabelMe.  Please let us know if you are thinking of using this.
We are curious about how you set the cost of the HIT, as well as the quality of the
annotations.  If you have any feedback on any part of the system
(instructions, annotation tool, etc.), please <a href="AboutMe.html" target="_blank">let us know</a>!
</p>




<h2>Advanced features</h3>

<p>
This section describes the structure of the <code>labelme.input</code>
file, which specifies the set of images to annotate
(the MATLAB function <code>generateLabelMeInputFile.m</code> can be
used to generate this file). The file starts
with the keyword <code>urls</code> and subsequently lists on each line
the URL of the annotation tool for each image to annotate.
The file has the following format:

<!--The following is a description of the structure of the
<code>labelme.input</code> file:-->

<!--The list of images to annotate needs to be added to the
<code>labelme.input</code> file.  The structure of the file is as
follows:-->
</p>

<p>
<table border="1">
<tr>
<td>
<code>
urls
<br />http://labelme.csail.mit.edu/tool.html?collection=LabelMe&amp;amp;mode=mt&amp;amp;folder=FOLDER&amp;amp;image=IMAGE1.jpg
<br />http://labelme.csail.mit.edu/tool.html?collection=LabelMe&amp;amp;mode=mt&amp;amp;folder=FOLDER&amp;amp;image=IMAGE2.jpg
<br />http://labelme.csail.mit.edu/tool.html?collection=LabelMe&amp;amp;mode=mt&amp;amp;folder=FOLDER&amp;amp;image=IMAGE3.jpg
</code>
</td>
</tr>
</table>
</p>

<p>
Make sure that each image is listed once.  This is important since
currently LabelMe does not handle concurrency, so multiple people
labeling the same image will overwrite each other.
</p>

<p>
We have added a few extra variables to the LabelMe annotation tool URL to customize the
annotation process.  Simply append <code>&amp;amp;VAR=VAL</code> to
the URL as needed.  The following is a list of variables:
</p>

<p>
<table border="1">
<tr>
<td align="center">Setting</td>
<td align="center">Meaning</td>
</tr>
<tr>
<td><code>&amp;amp;mt_sandbox=true</code></td>
<td>Use Mechanical Turk sandbox mode.  This mode is used for debugging
on Mechanical Turk.  You may want to start with this variable set to
make sure everything works.</td>
</tr>
<tr>
<td><code>&amp;amp;N=5</code></td>
<td>The worker is required to label at least 5 polygons.  Use
<code>N=inf</code> to allow the worker to label as many as they want.</td>
</tr>
<tr>
<td><code>&amp;amp;mt_intro=http://yourpage.com</code></td>
<td>You may customize the instructions that the worker sees.  By
default, the following <a href="mt_instructions.html" target="_blank">instructions</a>
are given to the workers.</td>
</tr>
<tr>
<td><code>&amp;amp;mt_instructions=Place your instructions here</code></td>
<td>
You may customize the one-line instructions that the worker sees at
the top of the labeling task. By default, the instructions are:
<b>Please label as many objects as you want in this image</b>.
</td>
</tr>
<tr>
<td><code>&amp;amp;actions=n</code></td>
<td>
This controls what actions the user is allowed to do.  The following
are possible actions:
<br /><br />
n - create and edit new polygons<br />
r - rename existing objects<br />
m - modify control points on existing objects<br />
d - delete existing objects<br />
a - allow all actions<br />
v - view polygons only (do not allow any editing)<br />
<br />
To set the desired actions, use any combination of the letters above.
For example, to allow renaming, modify control points, and delete
actions, then set <code>&amp;amp;actions=rmd</code>. By default,
<code>&amp;amp;actions=n</code>.
</td>
</tr>
<tr>
<td><code>&amp;amp;viewobj=e</code></td>
<td>
This will control which objects the user sees.  Use one of the following
possible options:
<br /><br />
e - view new and previously labeled objects<br />
n - view new objects only<br />
d - view new and deleted objects<br />
a - view all objects (new, existing, deleted)<br />
<br />
By default, <code>&amp;amp;viewobj=e</code>.  Note that for deleted
objects, these will be shown in gray and the object name in the object
list will be italicized.
</td>
</tr>
<tr>
<td><code>&amp;amp;objlist=visible</code></td>
<td>
This controls whether the object list on the right side is visible or
not.  Use <code>&amp;amp;objlist=hidden</code> to make it hidden.
</td>
</tr>
</table>
</p>

</td>
</tr>
</table>

<p>&nbsp;</p>
(c) 2009 MIT, Computer Science and Artificial Intelligence Laboratory.
<!--WEBBOT bot="HTMLMarkup" startspan ALT="Site Meter" -->
<p><script type="text/javascript" language="JavaScript">var site="s21labelme"</script>
<script type="text/javascript" language="JavaScript1.2" src="http://s21.sitemeter.com/js/counter.js?site=s21labelme">
</script>
</p><noscript>
<p><a href="http://s21.sitemeter.com/stats.asp?site=s21labelme" target="_top">
<img src="http://s21.sitemeter.com/meter.asp?site=s21labelme" alt="Site Meter" border="0"/></a>
</p></noscript>
<p><!-- Copyright (c)2005 Site Meter -->
<!--WEBBOT bot="HTMLMarkup" Endspan -->

</body>
</html>
